In the Classroom/En classe 


An Alternative Method for Teaching and 
Testing Reading Comprehension 

Robert Courchene 


The summary cloze technique, which can he used for both teaching and testing 
reading and listening comprehension, offers an alternative format to multiple- 
choice. Summary cloze exercises are prepared by summarizing the content of the 
original text into a new text about one third the length. The shortened text is then 
transformed into a rational cloze exercise. The learner, who always has access to 
the original text, must complete the summary text using the list of choices 
provided. Research using the technique has shown that it is certainly as good a 
measure of reading comprehension as the multiple-choice-based format and that 
it is often more reliable. Considerable experimentation, however, remains to be 
done concerning the text topic and choice and frequency of deletions. 


The most common method for testing (and frequently teaching) both listen¬ 
ing and reading comprehension consists of presenting a groups of students 
with a text followed by a series of multiple-choice (MC) questions to measure 
their understanding of what they have heard or read. The widespread use of 
this format is justifiable on many grounds—the tests can be machine or hand 
scored very easily allowing for the testing of large numbers of people in a 
short period of time; nonnative speakers can score the tests; well formulated 
MC questions can be used to test higher order language functions; and, 
finally, MC questions are reliable from a statistical point of view. Unfor¬ 
tunately, although the MC format is a desirable one, it is difficult to formu¬ 
late good questions consistently so that the reliability of a subtest does not 
become an issue. As well, questions could be raised concerning the face 
validity of the MC format—when one reads a text, not all processing is done 
in terms of possible choices. One frequently has to process material in other 
ways, for example, summarizing the information in a text to apply it to 
another context or to engage in problem solving activities. 

To find a more reliable and interesting format for testing reading com¬ 
prehension, Courchene and Raymond , 1 researchers at the Second Language 
Institute at the University of Ottawa, developed a new technique: the sum¬ 
mary cloze procedure. It was their intention that this testing/teaching proce¬ 
dure would be used as a backup and/or a replacement for MC questions to 
measure a candidate's ability to comprehend academic texts. In choosing the 
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cloze format, the authors were aware of the extensive and often contradic¬ 
tory research on the use of cloze as a measure of language proficiency in 
general and, more specifically, as a measure of reading comprehension 
(Oiler, 1979, 1983; Alderson, 1979; Bachman, 1985, Fotos, 1991, Klein-Braley 
& Raatz, 1984; Brown, 1989). Although the original claims for the cloze test as 
a measure of general language competence have been challenged, re¬ 
searchers still maintain that it is a good measure of language skills from the 
basic to the advanced levels as it correlates highly with reading comprehen¬ 
sion, dictation, and composition (Fotos, 1991). What has weakened the 
original claim has been evidence indicating (a) that rational and random 
deletion cloze tests vary in reliability and levels of difficulty, (b) that cloze 
tests often correlate more highly with sentence-level grammatical structures 
than higher order langauge skills, and (c) that not all deletions in a cloze test 
measure the same language abilities—they can measure both grammatical 
knowledge and textual relationships beyond the clausal level. Despite these 
difficulties, the cloze procedure, especially in its rational form, can be used to 
measure reading comprehension if one selects texts of general interest to 
students, uses a rational as opposed to a random deletion of items, pretests 
them on both native and nonnative speakers, and uses them in a foreign 
language context. 

In what follows, the summary cloze technique is described and demon¬ 
strated through the use of examples from reading; pedagogical applications 
are discussed; and finally a brief summary of the results of a study 
(Courchene & Ready, 1993) is presented. 

Summary Cloze: Selection and Preparation of a Text 
The first step in the preparation of a reading exercise or test is selecting a 
suitable passage. Although this will most commonly take the form of a 
standard text drawn from a newspaper, magazine, or book in the case of 
reading, it can also take the form of an advertisement, a notice, a set of rules 
for participating in a contest, and so forth. We have used the one-page 
Canada Food Guide, an advertisement from the National Film Board, an 
information sheet about a writing contest, along with more traditional texts 
from magazines such as MacLeans and Psychology Today. In every case we 
have been guided by the following general principles in preparing summary 
clozes. 

1. Select a text of an appropriate length, reading level, and interest for your 
students. In our studies we have worked with texts in the 450-500 word 
range (e.g., newspaper and magazine articles, but longer or shorter texts 
could also be used. In fact, the summary cloze can also be used as a 
summary of story that a child has read or possibly a chapter in a textbook 
for students at the secondary and postsecondary levels.) As our aim in 
using the technique both in the classroom and as a measure of reading 
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comprehension has been to teach or test global comprehension rather 
than the understanding of simple facts, texts that discuss concepts and 
ideas are best suited for use with this technique. Although I would not 
state categorically that one could not use this technique to teach students 
how to use a bus schedule, I think that other types of reading (e.g., skim 
and scan) would certainly be more appropriate. 

2. Once a text has been chosen, read it carefully and then set it aside; write a 
summary of the contents or list the most important ideas expressed in the 
text in your own words. Read the text again to be certain that you have 
the gist of the text. Next, identify the top level structure of the text or 
rhetorical pattern: description, sequence, causation, response, com¬ 
parison. In the written summary it is important to maintain the same 
rhetorical pattern as that found in the original text. If the original text was 
organized according to a causation textual structure, the summary should 
be organized in the same way. In preparing the summary one should use 
a balance of words and longer expressions found in the stimulus text 
along with suitable synonyms and alternative phrasing of ideas. By using 
large sections of the original text in the summary, one provides the 
students with more clues as to the correct word choices in the summary 
and thereby makes the summary cloze task easier to complete. This 
approach could be used if one wanted to use more difficult texts with low 
proficiency level learners. 

In terms of length, the summary for standard articles from newspapers 
and magazines should be about one third the length of the original. For 
other types of documents, such as advertisements, the summary in prose 
form should contain all the key information. 

3. Once the summary has been completed, select a number of words to be 
removed from the text. In doing so, use a rational deletion approach as 
opposed to every nth word. As one's purpose is to test overall under¬ 
standing of text content, the emphasis should be on the deletion of con¬ 
tent words, that is, deleting health as opposed to to in the phrase 
"contributing to the health of the nation"; and markers of textual 
coherence: removing because in "nursing women should drink sufficient 
quantities of milk because it helps to replenish ..." and so forth. As well, 
try to avoid words that can be predicted on the basis of knowledge of the 
world (although this is not always possible). Finally, avoid dates, proper 
names, and numbers unless they can be predicted on the basis of overall 
understanding of the text. It should be pointed out that the SC technique 
is not a disguised grammar test although being able to complete it 
presupposes some grammatical knowledge. 

In a summary passage of about 150 words, delete from 13-15 words. Do 
not decide in advance, however, that all texts must have 15 deletions and 
then go about creating them even when to do so results in some rather 
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questionable choices. Once the choices have been made, add other pos¬ 
sible word selections in the ratio of 5:4— for each four blanks, offer the 
students five possibilities making certain that none of the additional 
words can be used in the blank. The incorrect distracters in the list should 
be possibilities, however, from a grammatical and lexical point of view. 

4. The first and last sentences should not contain any blanks. As well, the 
title of the passage should also be included as it could activate a student's 
background knowledge. 

Sample Text 

Text: Changes in the Workplace 

When John Stevenson came home to tell his parents that he was quitting 
his job as the manager of a fast-food restaurant to enroll in secretarial 
school, he caused much anxiety and confusion in his family. The 
thought of their son becoming a secretary, a job that was usually held 
by a woman, was a shock to the family's pride. Ten years later, John is 
an executive secretary for a female vice-president. He is the pride of his 
family. All the negative feelings and attitudes associated with a man ac¬ 
cepting what had been traditionally seen as work performed by a 
woman have disappeared. 

John, however, has not been alone in his experience. Many women who 
have tried to enter jobs usually filled by men have had to prove them¬ 
selves every step along the way. For example, in the past, women who 
wanted to join the police force or become fire-fighters, had to meet the 
standards (height and physical strength) set for men in these profes¬ 
sions. As well, many women who wanted to join the military (army, 
navy, and air force), were refused many times before finally being ac¬ 
cepted by the military. The military kept insisting that they had wanted 
to protect women from the dangers of going to war. Once the women 
do get accepted in these positions, they find out that they have to per¬ 
form better than men to be considered as good. 

Both men and women who have decided to go for nontraditional jobs 
have found that the attitudes of other fellow workers change very slow¬ 
ly. This should not be surprising. At a very young age, boys and girls 
follow different sets of rules set by their parents and society. For ex¬ 
ample, it is quite acceptable for little girls to cry, but boys are not sup¬ 
posed to. On the other hand, if boys are angry, they can fight with each 
other, but this is not acceptable for girls. When these children become 
adults, they bring these attitudes, these ideas about what is expected of 
them to the workplace. If we want to change people's attitudes, we 
must change the way we bring up children. We must show them that 
boys and girls are capable of anything they want—from being president 
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of a country to exploring in Canada's North. What should decide their 
choice of profession should not be their sex but rather their own per¬ 
sonal interest and aptitude. (Text adapted from Ontario Times) 

Summary Cloze Texts 

Two possible summaries have been prepared. Version 1 at a lower level of 
difficulty and Version 2 at a higher level. The two versions were prepared 
using Fry's readability formula along with Bottel and Granowsky's (1972) 
formula for calculating syntactic complexity. Although such formulas do not 
ensure that texts that are similar in terms of these two traits will be equally 
difficult for all readers (other factors such as topic familiarity and cultural 
content also play an important role: Swaffar, Arens, & Byrnes, 1991; Gunder¬ 
son, 1991), they allow the teacher to control a certain number of variables. 

Version 1: Changes in the Workplace 

From the list of words provided, choose a suitable word for each blank space 
in the text below. You can use each word in the list only once. There are more 
words in the list than there are blanks or spaces in the text. 

Word List for Version 1 

(Place words in alphabetical order) 


t. 

attitudes 

5. difficulties 

9. people's 

13. sad 

2. 

better 

6. educate 

10. prepared 

14. secretary 

3. 

changed 

7. family 

li. president 

15. women 

4. 

combat 

8. military 

12 rules 

16. work 


Changes in the Workplace 

When John Stevenson decided to become a secretary, he shocked his 

whole family. They were not proud to have a son do female_. 

Ten years later, however, his_is proud of him; he is an executive 

_. But John is not unique. Many_who have tried to get 

jobs that are nontraditional have had many_, especially in the 

traditional male professions. At first, women in the_were not al¬ 
lowed in any roles that might involve_but this has finally 

changed. When they are accepted, however, women have to show they 

are_than men. Why has it been so difficult to change_? 

The answer is that people have different_for how boys and girls 

should behave in society. To change_attitudes, we must change 

the way we_our children. Only then will our children be able to 

make job choices based on personal interest rather than sex. 
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Version 2: Changes in the Workplace 
Word List for Version 2 


1 . 

acceptance 

9. employee 

17. members 

25. soldiers 

2. 

allowed 

10. encouragement 

18. overcome 

26. suffer 

3. 

case 

11. enroll 

19. position 

27. weight 

4. 

changed 

12. enter 

20. positive 

28. women 

5. 

concrete 

13. form 

21. possible 

29. worry 

6. 

combat 

14. impossible 

22. pressure 


7. 

deal 

15. male 

23. prevented 


8. 

dominated 

16. manager 

24. roles 



Changes in the Workplace 

Men and women who opt to work in professions or jobs not traditional¬ 
ly held by persons of their gender often encounter opposition. This op¬ 
position can take the_of social attitudes held by_of the 

society at large or_regulations and restrictions that almost make 

it_for them to qualify for the position in question. In the_ 

of John Stevenson, a managerial level_in a fast food restaurant 

who gave up his position to_in secretarial school, the opposition 

from his family took the form of negative_and shock. They felt 

that the family's pride would_by having a son in a female- 

_ profession. Fortunately for John, his family's attitudes have 

_over the last ten years. 

For women who have tried to_what have been until recently 

_-dominated professions—police, fire-fighters, soldiers 

— the road to gaining_has been more difficult. They have had 

to_opposition in terms of society's attitudes; i.e.,_can be 

in the military but they should not be_to undertake any combat 

roles. As well, women have had to_with concrete restrictions in 

the form of_and height requirements for certain professions. 

While there is no simple solution to helping men and women gain 
easier access to nontraditional professions, one must certainly start by 
trying to change how men and women are raised and perceived within 
society. 

Prior to using a summary cloze either as a test or an exercise in class, it is 
always a good idea to field test it with both native (friends and colleagues) 
and nonnative speakers. If native speakers have difficulty doing the text, or 
think that more than one word can be used in a given blank, make the 
necessary changes. With native speakers you might also try it out as an open 
summary cloze: give them the summary with the missing words and ask 
them to complete the text. Not only will this generate a good list of choices, it 
will also give you an idea of the content validity of the summary you have 
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prepared. Advanced nonnative speakers can also be asked to carry out the 
above exercises. With open-summary cloze exercises, some of the incorrect 
answers they suggest can eventually become some of the extra words that 
you add to your list. 

Pedagogical Suggestions 

1. When this type of exercise is used in class, the students should be en¬ 
couraged to take their time. They should be told that they can always refer 
back to the original text while trying to complete the summary. If it is 
used in a testing situation, more time has to be given than for passages 
with MC questions because you are asking the students to undertake a 
double reading task. More processing is required to complete the sum¬ 
mary cloze task than to answer MC questions. 

2. As indicated above, it is possible to use the summary cloze technique in 
different ways: 

a. reading passage followed by a summary cloze with words provided; 

b. reading passage followed by an open summary cloze in which 
students have to supply their own words; 

c. reading passage followed by two or more versions of a summary 
cloze at different levels of difficulty and/or with different response 
formats (words provided/not provided). 

Research Study 

To determine how SC functions as a measure of reading ability when com¬ 
pared with the MC format, and to determine how SC correlates with other 
measures of language proficiency (listening, reading, skimming and scan¬ 
ning and canonical cloze), Courchene and Ready (in press) 2 conducted a 
study involving students preparing to come to Canada for academic and 
professional training. For the study, five reading texts controlled for length, 
syntactic difficulty, and reading level were prepared with both MC questions 
and SC formats. The subjects were 66 Chinese students at an intermediate to 
advanced level matched for language ability and then randomly assigned to 
one of two groups. Each group did two texts in one format and two in the 
other and both groups did both the SC and MC formats for the fifth text. 
After reading each text the subjects completed a Reader Assessment Ques¬ 
tionnaire focusing on text difficulty, background knowledge, and interest. 

From the results it was evident that the SC technique produces tests that 
tend to yield higher levels of reliability than their MC counterparts. For three 
out of five of the texts the mean scores for the two tasks were not significantly 
different from each other although in all cases the mean of the SC task is 
somewhat lower. The significant differences in mean scores for the first two 
texts could be accounted for by the fact that the SC task was unfamiliar; it 
was the first SC exercise done by each group. The fact that the MC group 
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reported that they found that the first text easier than did the SC group is also 
probably related to their familiarity with the testing format. 

It is worth noting that in spite of the fact that the texts were chosen to be 
at the same readability level, the first text was much more difficult than the 
others. This may be because the topic (the question of food additives) would 
be unfamiliar to the Chinese students. Thus readability is not the only 
criterion to consider when choosing texts for development as ESL tests or 
classroom exercises. 

The correlations of the task types with general measures of ESL profi¬ 
ciency do provide evidence that there are no substantial differences in the 
way the tasks behave, and in general the assumption holds that the two task 
types are both measures of reading comprehension. What is also interesting 
to note is that the correlation patterns vary widely from one text to another. 
There is evidently a great deal of variability in the factors that determine 
success in responding to a text about which at this time the item writer has 
very little knowledge and, therefore, cannot control. 

In general, the results support the author's contention that the SC tech¬ 
nique is a viable and very often more reliable alternative to the MC format 
when testing reading comprehension. What strategies or skills we are actual¬ 
ly testing under the aegis of reading comprehension are not so obvious from 
these results. It would probably be useful in the context of this study to 
explore via protocol analysis how student approaches differ when con¬ 
fronted with MC and SC tasks, and how the SC technique differs from or 
resembles the canonical cloze procedure. Evidently a great deal more re¬ 
mains to be understood and in this we echo Alderson (1991) who states, "we 
have much more work to do before we can claim to have identified what it is 
that any test tests, and therefore are equally far from being able to claim that 
we have perfected our tools for test analysis" (p. 12). The SC technique does, 
however, provide an alternative testing method for the measurement of 
reading comprehension. 

Conclusion 

The summary cloze technique was developed to offer language teachers and 
test constructers another possible format for both teaching and testing read¬ 
ing comprehension. Initial use in the classroom and as a testing instrument 
have resulted in positive feedback from the students. They see summary 
cloze as having face validity; they feel that it is a legitimate and effective way 
to become better readers and to test what they have learned. Finally, sum¬ 
mary cloze should not and cannot replace MC questions as the only accept¬ 
able way to test reading comprehension. As with any other teaching 
technique, it should be used judiciously in both the classroom and the testing 
hall: cranking out summary cloze exercises according to some preconceived 
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formula will reduce it to a type of mindless drill and, as a result, drain it of 
any possible pedagogical value. 

Notes 

lr The technique was originally developed in the context of test development for students in 
comprehension-based courses. On one of the tests, students had been given an advertisement 
from the National Film Board of Canada along with a series of True/False questions. Even 
though the subsequent statistical analyses indicated that the True/False questions were not 
tapping the content of the text, the test developers still wanted to include such reading texts as 
part of the test bank. As this point, one of the researchers decided to write a summary of the 
information in the text and to turn it into a summary cloze. When testing indicated that the 
technique was reliable, the two researchers then developed a whole battery of texts that were 
used for teaching and testing (Courchene & Ready, in press). 

2 For a complete discussion of this study see Courchene and Ready (in press). 
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